Input device using scanning sensors

ABSTRACT

An electronic device receives inputs using images of input devices and scanning sensors that detect user interaction with those images. For example, an image of a mouse may be projected and the user&#39;s interaction with that image may be scanned using digital video cameras. These user movement images may then be analyzed to determine, for example, the direction, extent and rate of movement of the user to provide pointer position command signals. Similarly the user&#39;s movement relative to a virtual keyboard image may be analyzed and converted into key actuation signals for responding to the actuated virtual keys.

This application is a continuation-in-part of application Ser. No.09/167,814, filed on Oct. 7, 1998.

BACKGROUND

This invention relates generally to input devices for electronic devicesincluding computer systems and particularly to such devices using videocontrol inputs.

A variety of pointing devices have been used in computer systems tocontrol the position of a pointer or cursor on a display. A pointer orcursor is an image that may be moved across the screen to select aparticular option provided in the form of a physical depiction on a userinterface. In this way, the user may interact with a video display toprovide focus to particular tasks and activities offered through theuser interface.

Known pointing devices include the mouse, track ball and the pointingstick. All of these devices generally involve an electromechanicalinterface which is prone to failure. When the interface fails, it mayresult in an incorrect signal which may cause movement of the pointingdevice to be incorrectly reflected in the pointer position of thedisplay.

Another well known pointing device, the touch pad pointing device,allows the user to touch a contact surface. Movements of the user'sfinger on the contact surface are tracked as corresponding movements ofthe pointer on the computer display. Normally, touch pads havecapacitive sensors which detect finger position and movement andtranslate that movement into position control signals.

The touch pad pointing devices are subject to being fouled by dirt ormoisture, resulting in incorrect signals as described above. In additionthe touch pad devices provide a limited degree of movement, making itdifficult to learn to control the pointer using the limited movementsavailable through the pointing devices.

A number of computer systems use keyboards for inputting textinformation to the computer. The keyboard is generally connected to thecomputer by a cable. Like the pointing device, the keyboard involves anelectromechanical interface which is prone to failure. These failuresmay result in the inability of a particular key to activate a particularinput. In any case, inputs which are provided to the keyboard may beinaccurately reflected in the text which appears on the display screen.

Thus, there is a need for an improved input device which allows the userto input information in ways which may be less prone to failure.

A number of electronic devices, including computer systems, includerelatively bulky input devices such as keyboards. For example, in manyportable computers such as laptop or notebook computers, a foldinghousing is used in which the display is essentially one portion of theoverall housing and the keyboard is the other portion. Thus, aconsiderable portion of the overall bulk of the device is taken up bythe keyboard. Similarly in some personal digital assistants (PDAs) orpalm-sized computers, the keyboard or keypad input device takes up aconsiderable portion of the overall bulk of the device.

There is a demand for ever smaller computing devices. Portable computerssuch as laptop or notebook computers are progressively decreasing insize. Similarly there is increasing interest in palm-sized or PDA-typecomputer systems.

Thus, there is a continuing demand for new ways to reduce the size ofelectronic devices including computer systems.

SUMMARY

In accordance with one embodiment, a method for inputting data to anelectronic device includes projecting an image on a surface. Usermovements are detected relative to the projected image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a side elevational view of one embodiment in accordance withthe invention;

FIG. 2 is a front elevational view of the embodiment shown in FIG. 1;

FIG. 3 is a top plan view of the images which may be projected by thedevice shown in FIG. 1;

FIG. 4 is a block diagram of one embodiment in accordance with theinvention for implementing the device shown in FIG. 1;

FIG. 5 is a flow diagram of the process of forming a device which canprovide the features of the device shown in FIG. 1 in one embodiment inaccordance with the invention;

FIG. 6 is a flow diagram for one software implementation of theembodiment shown in FIG. 1; and

FIGS. 7 and 8 are flow diagrams for calibration software in oneembodiment in accordance with the invention.

DETAILED DESCRIPTION

An electronic device which may be implemented as the computer system 10,shown in FIG. 1, may include a housing 12 which includes a display 18.Unlike conventional computer systems, the system 10 may not have akeyboard or a pointing device. Scanning sensors 14 and 16 may beprovided on extended arms 26 and 30 to enable scanning of the region infront of the housing 12. The scanning sensors 14 and 16 may, forexample, be digital cameras that produce streaming video which isconducted through cables contained in the arms 26 and 30 to the housing12. An additional scanning sensor 22 may be provided on the front faceof the housing 12 beneath the display 18, as shown in FIG. 2.

Also situated on the housing 12 is a projector 20 that projects an imageonto a surface, such as a desktop, situated in front of the housing 12.The projector 20 may for example, be a liquid crystal display (LCD)projector, a laser projector or a light engine projector such as adigital light processing (DLP) light engine from Texas Instruments, Inc.The region where the image is projected may be the same region which isscanned by the scanning sensors 14, 16 and 22. Each scanner may bealigned transversely to each of the other scanners for better threedimensional imaging. The housing 12 may be supported at an angledorientation by the stand 24.

In one embodiment in accordance with the invention, one or more inputdevices, shown in FIG. 3, such as a virtual keyboard 34 a and mouse 34 bmay be projected onto a surface in front of the housing 12. However, theinvention is not limited to this illustrated embodiment. When the userpositions his or her hands over the image as the user would in using aphysical input device, the scanning sensors may capture the movement ofthe user's hands and convert this movement into input command signals.For example, movements of the user's hand over the surface upon whichthe mouse image 34 b was projected may be converted into pointerposition signals for moving the position of the pointer on the display18. Similarly, typing movements of the user's hands over the surfaceupon which the keyboard image 34 a was projected may cause thecorresponding textual input to be entered and to appear on the display18. Thus, an electromechanical input device may be eliminated.

Alternatively, the scanning sensors may sense the projected lightreflected from the user's hands. The scanning sensors may also detectthe combination of the reflected, projected light and the user's hands.Also, the projected images may move so as to “follow” the detectedmovement of the user's hands.

Turning now to FIG. 4, an example of a computer system 35 forimplementing one embodiment in accordance with the invention may includea processor 36 coupled to a host bus 38. The illustrated computer system35 includes the 440BX chip set with a Pentium® II processor availablefrom Intel Corporation. The host bus 38 is coupled to a host bridge 40which in turn couples a main memory 42. The host bridge 40 also couplesan accelerated graphic port (AGP) bus 48 (see Accelerated Graphics PortInterface Specification, Revision 1.0, published on Jul. 31, 1996, byIntel Corporation of Santa Clara, Calif.) which couples a graphic device50. The graphics device 50 may receive inputs from the scanning sensors14, 16 and 22 and may display the resulting inputs on a display 18 whichis also coupled to the graphics device 50. The graphics device 50 mayalso have a video basic input/output system (BIOS) 54 and a graphicslocal memory 56.

The host bridge 40 is also coupled to a bus 44 which may include aplurality of slots 46 for receiving peripheral devices. The bus 44 isalso coupled to a bus-to-bus bridge 58 which is in turn coupled to anadditional bus 70. The bus 70 includes a plurality of slots 72 forreceiving additional peripheral devices and the system BIOS 74. Thebridge 58 may also couple to a plurality of ports 60 and 66. The port 60may be coupled, for example, to a hard disk drive 62 which stores one ormore programs 64, 106 and 122 as indicated. The port 66 may. be coupledto the projector 20.

While one example of a computer system is shown in FIG. 4, those skilledin the art will appreciate that a number of other systems may beutilized to achieve the objectives set forth herein. Therefore it shouldbe understood that the system illustrated is only an example.

Turning now to FIG. 5, one embodiment of a method for providing thedevices having the previously described functionality is illustrated inaccordance with the invention. Initially a system is provided with videocapability, as indicated in block 76. For example, a computer system maybe provided with the ability to process a plurality of streaming videoinputs. The system may also be equipped with a plurality of digitalcameras. A projector is enabled to project an image of an input deviceon a surface, as indicated in block 78. The ability to provide videoanalysis of the resulting movements of the user's hands, for example,may also be provided as indicated in block 80. In accordance with oneembodiment, the video analysis may be a conventional pattern recognitionsoftware which recognizes the shape of the user's hand and correspondingmovements thereof. Next, the ability to convert the user's movements inassociation with the projected images to a pointer position controlsignal may be provided, as indicated in block 82. Finally, the abilityto convert the movements of the user into text inputs may be provided,as indicated in block 84. In addition, an audible confirmation, such asa key clicking sound may be produced when a keystroke is recognized.

Turning now to FIG. 6, software 64 may be stored on the hard disk drive62 in one embodiment according to the invention. Initially the softwarecauses the image of one or more input devices to be projected onto asurface. For example, the image of a keyboard and a mouse may beprojected, as indicated in block 86. Thereafter streaming video may bereceived (block 87) and a particular video frame may be selected, asindicated at block 88. The frame may be selected randomly, or at regularintervals, for example, using a selection technique.

In addition, a scene change detector may be used to determine when thereis a significant change in scene. When a significant scene changeoccurs, a frame may be captured for subsequent analysis. Scene changedetectors are known in the art and may involve an analysis of thepattern of intensity signals determined by the pixels that make up animaging array in one or more cameras. For example, the various pixelsensors may provide output signals. If an output signal is above acertain intensity level, it may be considered white and if it is below acertain level it may be considered to be black. A significant change inthe number of white versus black output signals can be recognized as ascene change initiating the selection of one frame from the streamingvideo.

While the present invention has been described with respect to the useof a single frame for this analysis, a continuing analysis can be doneby successively or continually analyzing successive frames. oversamplingmay be used to insure that movements are detected.

In the embodiment illustrated in FIG. 6, once a frame is selected, asindicated in block 88, conventional pattern recognition software may beutilized to determine if a particular image pattern is represented bythe digital video data stream, as indicated in block 92. This softwaremay recognize a particular preprogrammed image such as the position ofthe user's fingers on a virtual keyboard or the position of the user'shand on top of a virtual mouse. This software may recognize the image bycomparing it to a previously recorded depiction or the software may betrained to learn a particular image through cooperation of the computerowner or user, using learning software.

At diamond 94, the software checks to determine whether the selectedframe contains a pattern matching a predetermined pattern whichcorresponds to a particular input command. If not, flow returns toselect another frame.

If a pattern match is detected, the first and a subsequent frame arecompared as indicated in block 96. Next, a check determines whether thecomparison indicates a change in position (diamond 98). If so, the rateand direction of movement of the image may be determined (block 100).The rate and/or direction of movement can be used to providecorresponding command inputs. For example, in connection with themovement of the user's hand in association with the mouse image, thedirection and rate of movement can be reflected in correspondingmovements of the pointer on the display 18. For example, a correlationfactor may be used to calculate the three dimensional position of an “onscreen” pointer (block 102). The user inputs may be represented by asignal to the graphics device 50 to move the pointer on the display 18according to the direction or rate of movement of the user's hand (block104). The flow cycles back to select a new frame (at block 88) to startthe comparison again.

Alternatively, a distinct object may be associated with the user so thatthe object may be easily recognized. For example, a distinctly coloredring may be used which can be readily detected. Movement of the ring maybe detected as a pointer position command.

Software 106 for enabling the system.to calibrate a focus point forpointer position commands is illustrated by FIG. 7. The software 106 maybe utilized during a calibration period in order to program the systemto recognize the desired inputs. Referring to FIG. 7, initiallystreaming video is received as indicated in block 108. A frame of thatstreaming video is captured, as illustrated in block 110.

Next, the system is prompted for movement in a defined direction asshown in block 112. For example, an on screen display may guide the userthrough the learning process. For example, the on screen display mayfirst direct the user to show a left cursor position command. Thesequence then cycles through each of the cursor command directions. Oncethe system knows what cursor command is being programmed, it may capturea new frame of videoas indicated in block 114. The new frame and theprior frame are compared to determine a difference that amounts to agiven cursor command (block 116). A check atdiamond 118 determineswhether a focus point has been identified. In other words, the systemmust be able to differentiate the two frames sufficiently to be able tothereafter recognize those differences as a particular pointer positioncommand. If not, the process is tried again. Otherwise, the system savesoff the focus point data of the identified focus point, as indicated inblock 120.

Referring to FIG. 8, the software 122 enables a calibration of speed anddirection of pointer position commands. The focus point data determinedby the software 106 is loaded at block 123. Streaming video is receivedas indicated at block 124 and a frame is captured as shown in block 126.A focus point is identified such as the user's finger as indicated inblock 128 and a movement prompt is provided (block 130). The prompt maybe a message block on the screen asking the user to implement what theuser wants to use as a desired movement. A new (subsequent) frame iscaptured in block 132 and compared to the frame captured in block 126,as illustrated by block 134.

The focus point movement is identified (block 136) and a direction andrate of movement are calculated as indicated in block 138. A correlationfactor for x, y and z directions is used to calculate the correspondingrate and direction of movement of the screen pointer (block 140). Thescreen pointer is then moved as indicated in block 142. The user thenmay adjust the amount of pointer movement for a given movement of theuser's finger, for example through inputs provided by a prompt screen,as indicated in block 144. In addition, the user may provide alternativeinputs which could be received as a particular pointer position command.The user is then prompted to indicate whether the user is done indiamond 146. If not, the flow cycles and new commands may be learned asdescribed above. Otherwise the flow is terminated and the information issaved (block 148).

The number of cameras that may be used may be subject to considerablevariation. More cameras may result in better three dimensional imaging.This may allow movements in three dimensions to be recognized as inputcommand signals. In addition, camera redundancy may be used toeliminated errors.

While the present invention has been illustrated with respect toseparate cameras mounted on a computer, the cameras could be integratedinto the housing of the computer system itself. In addition, while theinvention has been illustrated in connection with cameras sensitive tolight in a visible spectrum, infrared detecting cameras could be used aswell.

By using streaming video to control input information, anelectromechanical connection (which may be prone to errors or failure)may be eliminated in some embodiments. In addition, the accuracy of theinputs may be improved in some embodiments, for example, as compared totouch pad pointing devices, because the possibility of incorrectactuation due to dirt or moisture may be eliminated.

Other input devices may be handled in the same fashion. For example, atelephone may be replaced with the projected image of a telephone. Acomputer microphone may be activated when the handset is grasped andlifted as recognized by the scanning sensor system. Dialing could behandled in the same fashion described previously in connection withactuation of the keyboard projected image.

The projected image may be altered to correspond to the aportion of theinput device the user sees with his or her hands positioned over anormal input device. This altered image may be produced when the user'shands are detected as being in position over the surface that receivedthe projected image. Moreover, the input device image may itself bemodified to reflect the user's interaction with that image. For example,when finger movement corresponds to key depression, the projected imagemay be altered to suggest that the virtual key has moved.

While the present invention has been described with respect to a limitednumber of embodiments, those skilled in the art will appreciate numerousmodifications and variations therefrom. It is intended that the appendedclaims cover all such modifications and variations as fall within thetrue spirit and scope of the present invention.

What is claimed is:
 1. A method for inputting data to an electronicdevice comprising: projecting an image of a mouse on a surface; anddetecting user movements relative to said mouse image.
 2. The method ofclaim 1 further including capturing a digital video image of the user inassociation with the projected image and inputting a representation ofsaid image to the electronic device.
 3. The method of claim 2 furtherincluding analyzing said digital video image including using patternrecognition software.
 4. article comprising a medium for storinginstructions that, if executed, enable a computer to: project an imageof a mouse on a surface; and detect user movements relative to saidimage.
 5. The article of claim 4 further including instructions thatcause a computer to analyze a digital video stream using patternrecognition software.
 6. A system comprising: a processor; a sensorcoupled to said processor; a projector coupled to said processor; and amemory associated with said processor, said memory storing a computerprogram that enables a processor to project an image of a mouse usingsaid projector and use said sensor to detect user movement relative tosaid image of said mouse.
 7. The system of claim 6 including a pluralityof digital video cameras.
 8. The system of claim 6 wherein said systemis a computer system.
 9. The system of claim 6 wherein said memoryincludes pattern recognition software.